Speech-to-text API

Implement the speech-to-text (STT) API to act as an STT provider for Live Hub, whether for a recognition service of your own or for a vendor Live Hub does not integrate with directly. Live Hub is the client: it streams the caller's audio to you, and you return what was said.

Register the service as a custom integration speech provider, and enter your endpoint in the 'Speech to Text (STT) URL' field, in the form ws://example.com/api/v1/speech:recognizeASR.

How the connection works

Live Hub opens one WebSocket per conversation and keeps it open for the whole call, including periods when no recognition is taking place at all.

That connection carries recognition sessions, one for each stretch of audio to recognize. They run one after another, never at the same time: the audio arrives as binary frames, and a binary frame carries no session information, so a connection can have only one recognition session active at a time. When a session ends, the next can start on the same connection.

Control messages are JSON text frames. Audio travels as binary frames, per section 5.6 of RFC 6455.

If an error leaves your service unable to handle further messages on the connection, close the connection.

Authentication

Live Hub sends the shared token from the provider's 'Authentication Key' field in the Authorization header of the HTTP request that opens the WebSocket:

Authorization: Bearer {token}

See Security and authentication.

Configuration

One parameter controls the audio format Live Hub announces to your service:

Parameter Type Description
sttPreferWave boolean Which audio format Live Hub announces in format. true sends WAV, with headers. false sends RAW, without.

Messages from Live Hub

Live Hub sends the following messages to your service.

start

Live Hub sends start to begin a recognition session.

Parameter Type Description
language string BCP-47 language code to recognize the audio in.
conversationId string ID of the conversation.
format string raw for audio without headers, wav for audio with WAV headers. Follows sttPreferWave.
encoding string How the audio is stored and transmitted. Only 16-bit linear PCM, LINEAR16, is supported.
sampleRateHz number Sample rate of the audio, in hertz. Only 16000 is supported.
sttContextId string The value of the bot's sttContextId parameter, when one is configured.
sttSpeechContexts array The value of the bot's sttSpeechContexts parameter, when one is configured.
sttGenericData string The value of the bot's sttGenericData parameter, when one is configured.
participant string Which participant the audio belongs to, on agent-assist calls. 
{
  "type": "start",
  "language": "en-US",
  "conversationId": "8745555-8f1a-48ba-9ec9-46e90dc5aa18",
  "format": "raw",
  "encoding": "LINEAR16",
  "sampleRateHz": 16000
}

stop

Live Hub sends stop to end the current recognition session. It sends stop only for a session you have already acknowledged with started.

{
  "type": "stop"
}

Audio

Between start and stop, the audio arrives as WebSocket binary messages, in the encoding and at the sample rate the start message announced.

Messages from your service

Your service sends the following messages to Live Hub.

started

started reports that the recognition session is running and your service is ready for audio.

{
  "type": "started"
}

hypothesis

hypothesis carries a partial result, sent as recognition progresses.

{
  "type": "hypothesis",
  "alternatives": [
    {
      "text": "Hi"
    }
  ]
}

recognition

recognition carries a recognized utterance. Send one per utterance; a single recognition session can produce several.

{
  "type": "recognition",
  "alternatives": [
    {
      "text": "Hi there",
      "confidence": 0.8355
    }
  ]
}

end

end reports that the recognition session has ended. Send it after a stop, to confirm the session is over. If your service recognizes only one utterance per session, send it immediately after the recognition message.

{
  "type": "end",
  "reason": "some reason"
}

error

error reports that the recognition session ended in failure.

{
  "type": "error",
  "reason": "some error"
}

Example session

The following exchange recognizes two utterances in one session, and then opens a second session on the same connection.

  1. Live Hub → you — start the session:

    {
  "type": "start",
  "language": "en-US",
  "conversationId": "8745555-8f1a-48ba-9ec9-46e90dc5aa18",
  "format": "raw",
  "encoding": "LINEAR16",
  "sampleRateHz": 16000
}

  2. You → Live Hub — ready:

    {
  "type": "started"
}

  3. Live Hub → you — the audio, as binary frames.

  4. You → Live Hub — a partial result:

    {
  "type": "hypothesis",
  "alternatives": [
    {
      "text": "Hi"
    }
  ]
}

  5. You → Live Hub — the first utterance:

    {
  "type": "recognition",
  "alternatives": [
    {
      "text": "Hi there.",
      "confidence": 0.8355
    }
  ]
}

  6. You → Live Hub — the second utterance:

    {
  "type": "recognition",
  "alternatives": [
    {
      "text": "My name is John.",
      "confidence": 0.83
    }
  ]
}

  7. Live Hub → you — stop:

    {
  "type": "stop"
}

  8. You → Live Hub — session over:

    {
  "type": "end",
  "reason": "stop by client"
}

Live Hub can now send another start on the same connection, and the cycle repeats.